Regression: Cinder Volumes "unable to find iscsi target" for VMware instances

Can you re-run this example with the logging level set to 'debug' for us?
 I need to see which part of this branch gets followed:

        if device_name:
            LOG.debug(_("Discovered iSCSI target %(target_iqn)s from "
                        "%(target_portal)s."),
                      {'target_iqn': target_iqn,
                       'target_portal': target_portal})
        else:
            LOG.debug(_("Unable to discovered iSCSI target %(target_iqn)s "
                        "from %(target_portal)s."),
                      {'target_iqn': target_iqn,
                       'target_portal': target_portal})

... these only log in debug mode.

Hey Shawn,

Sorry for the delay on this.

Here is the error output with debug and verbose set to True in nova.conf on the nova-compute node, hope this is correct, if not let me know I can dump more output:

http://paste.openstack.org/show/42932/

@sina-sa

Obviously, I'm taking a little bit to come back around on some of these myself.

I thought I had written you a reply before, but I must have forgotten to hit "Post Comment"

The first 11 lines are the interesting part.

> Unable to discovered iSCSI target iqn.2010-10.org.openstack:volume-d0433ba1-d2c1-478c-96a1-b9074c57c62e from 192.168.125.27:3260. discover_st /usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py:111

Means that 192.168.125.27 canot reach iqn.2010-10.org.openstack:volume-d0433ba1-d2c1-478c-96a1-b9074c57c62e so the driver can't complete the action. To further diagnose this ... we need to figure out how to tell if that network segment really can be traversed or not.

Can you verify you can make the network traversal?

Hey Shawn,

Again sorry for the delay.

From the nova-compute host I can definitely connect to the ip and port

root@nova-compute:~# nc -v 192.168.125.27 3260
Connection to 192.168.125.27 3260 port [tcp/*] succeeded!

(for reference, the .27 is the cinder-api/volume/scheduler host and .28 is the nova-compute host)

Hi Sina,

Could you please post your cinder and nova configuration? Also, could you please describe your cinder configuration including:

1. What cinder driver are you using?
2. Are you trying to mount an LVM volume served by iSCSI?
3. Are you trying to mount an external Cinder node via iSCSI?

Thanks!

Hi Ryan,

1. Cinder driver is default in this case
2+3. Not sure I understood these questions? When you say mount, what are you referring to? When you say "external cinder node" what are you referring to?

In nova.conf on the nova-compute node the only configuration option specified for volumes is:
volume_api_class=nova.volume.cinder.API

In cinder.conf the related options (which are default):
iscsi_helper=tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes

The cinder server is running cinder-api, cinder-volume, cinder-scheduler along with a separate server running nova-compute.

I think root cause is here:

https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/volumeops.py#L172

def get_volume_connector(self, instance):
        """Return volume connector information."""
        instance_name = instance['name']
        vm_ref = vm_util.get_vm_ref(self._session, instance)
        iqn = volume_util.get_host_iqn(self._session, self._cluster)
        return {'ip': CONF.vmware.host_ip,
                'initiator': iqn,
                'host': CONF.vmware.host_ip,
                'instance': vm_ref.value}

The 'host' is presumed to be the SDK host. In the case of vCenter, the host will be the ESXi host the instance is run on.

I was able to reproduce the error message, "StorageError: Unable to find iSCSI Target", using the latest upstream code. You need to be using the vSphere nova driver and the default cinder driver. Steps to reproduce:

1. Spawn an instance
2. Create a volume
3. Attach the volume to the instance

At this time, monitor n-cpu.log and you'll see the error:

[req-b76ae0c1-2cd3-4794-8b52-747abe65351e demo invisible_to_admin] Exception during message handling
Traceback (most recent call last):
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
    **args)
  File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)
  File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
    payload)
  File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
    return f(self, context, *args, **kw)
  File "/opt/stack/nova/nova/compute/manager.py", line 243, in decorated_function
    pass
  File "/opt/stack/nova/nova/compute/manager.py", line 229, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 271, in decorated_function
    e, sys.exc_info())
  File "/opt/stack/nova/nova/compute/manager.py", line 258, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 3577, in attach_volume
    context, instance, mountpoint)
  File "/opt/stack/nova/nova/compute/manager.py", line 3572, in attach_volume
    mountpoint, instance)
  File "/opt/stack/nova/nova/compute/manager.py", line 3614, in _attach_volume
    connector)
  File "/opt/stack/nova/nova/compute/manager.py", line 3604, in _attach_volume
    mountpoint)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 611, in attach_volume
    mountpoint)
  File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 274, in attach_volume
    self._attach_volume_iscsi(connection_info, instance, mountpoint)
  File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 247, in _attach_volume_iscsi
    raise volume_util.StorageError(_("Unable to find iSCSI Target"))
StorageError: Unable to find iSCSI Target

If I'm right about root cause this might fix it:

* https://review.openstack.org/45906

It's a one-liner.

Still not clear on root cause for this.

I guess you are hitting this issue because the iscsi server location from where the iscsi tagets are discovered is not added to the iscsi software adapter on the host. When I added the iscsi server location, I was able to attach the cinder volume as a rdm disk :)

Having said that, it's the driver that should add the iscsi server address to the adapter (provided the adapter has Send Target Discover method enabled) and do the rescan.

I have a small patch to get this going. I will soon upload it after adding some tests.

Fix proposed to branch: master
Review: https://review.openstack.org/49465

VMware drivers fail to attach a iSCSI backed cinder volume because the underlying retrieval api was updated to a newer version (see bug 1183654) but not it's usage in volume_util. This is a regression over Grizzly release. Also, we fail to specify the iscsi server location (target portal) in the api calls. Both are critical for the existing feature to work.

Reviewed: https://review.openstack.org/49465
Committed: http://github.com/openstack/nova/commit/e70211ff89785fed6a868eaab723847833590502
Submitter: Jenkins
Branch: master

commit e70211ff89785fed6a868eaab723847833590502
Author: Sabari Kumar Murugesan <email address hidden>
Date: Thu Oct 3 23:50:05 2013 -0700

    VMware: fix regression attaching iscsi cinder volumes

    VMware drivers fail to attach a iSCSI backed cinder volume with an
    error 'unable to find iscsi targets' because the underlying
    retrieval api was updated to a newer version but not it's usage in
    volume_util (see commit e331287cdd1300878bfb9bbb3630889370e5fa72).
    This is a regression over Grizzly release.

    Fixes: bug #1193980

    Change-Id: Ic2d1dc800fa10295a4d7312fbeeaa0e16d792755

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/53595

Reviewed: https://review.openstack.org/53595
Committed: http://github.com/openstack/nova/commit/66f2db53460520019c6845be7f2997d6769dfe0d
Submitter: Jenkins
Branch: stable/havana

commit 66f2db53460520019c6845be7f2997d6769dfe0d
Author: Sabari Kumar Murugesan <email address hidden>
Date: Thu Oct 3 23:50:05 2013 -0700

    VMware: fix regression attaching iscsi cinder volumes

    VMware drivers fail to attach a iSCSI backed cinder volume with an
    error 'unable to find iscsi targets' because the underlying
    retrieval api was updated to a newer version but not it's usage in
    volume_util (see commit e331287cdd1300878bfb9bbb3630889370e5fa72).
    This is a regression over Grizzly release.

    Fixes: bug #1193980

    (cherry picked from commit e70211ff89785fed6a868eaab723847833590502)

    Change-Id: I14fe62ad98fd9161df3ce6c06160b2759a9e38b2